DPO cleanup #1126
Merged
DPO cleanup #1126
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JohanWork
reviewed
Jan 16, 2024
plaguss
reviewed
Jan 18, 2024
plaguss
reviewed
Jan 18, 2024
There was a problem hiding this comment.
Awesome PR! I left a comment in case you see fit. Also, maybe it could be tackled in a different PR, but the preprocess command could also be updated to allow checking rl datasets:
+ if parsed_cfg.rl:
+ _ = load_rl_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
+ else:
+ _ = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
- _ = load_datasets(cfg=parsed_cfg, cli_args=parsed_cli_args)
plaguss
reviewed
Jan 18, 2024
winglian
force-pushed
the
dpo-cleanup
branch
2 times, most recently
from
d5f97c3 to
c0a1553
Compare
filippo82
reviewed
Jan 23, 2024
|
|
||
| def load(strategy, cfg): | ||
| try: | ||
| load_fn = strategy.split(".")[-1] |
There was a problem hiding this comment.
This is most likely not correct. The strategy includes underscores, not ., such as intel_apply_chatml.
There was a problem hiding this comment.
def load(strategy, cfg):
try:
load_fn = strategy.split("_")[-1]
#strategy = ".".join(strategy.split("_")[:-1])
LOG.info(load_fn)
LOG.info(strategy)
mod = importlib.import_module(f".{load_fn}", "axolotl.prompt_strategies.dpo")
func = getattr(mod, strategy)
load_kwargs = {}
return func(cfg, **load_kwargs)
except Exception as e: # pylint: disable=broad-exception-caught
LOG.warning(e)
return NoneThere was a problem hiding this comment.
the intention is the setting is something like
type: chatml.argillain which case it will load the argilla function from the axolotl.prompt_strategies.dpo.chatml module.
There was a problem hiding this comment.
Hi @winglian 👋🏻 thanks. That makes sense. I will test it later today 👍🏻
filippo82
reviewed
Jan 23, 2024
Co-authored-by: Agus <[email protected]>
Co-authored-by: Agus <[email protected]>
djsaunde
pushed a commit
that referenced
this pull request
Dec 17, 2024
* cleanup dpo to be a little more extensible, add zephyr/nectar strategy * fix eos slash * support for eval split * fix kwargs * handle empty evals * don't load peft model for dpo * ensure dpo traning args gets bf16 for peft if applicable * fix duplicate kwargs for bf16 * make sure to respect the configured lr scheduler * supprt trainer callback to push config to wandb * set dataloader preload args * ensure that we are loading the lora when merging * Update src/axolotl/utils/data.py Co-authored-by: Agus <[email protected]> * support local datasets for dpo Co-authored-by: Agus <[email protected]> * chore: lint * dpo/kto/ipo smoke tests w lora, simplify dpo dataset type names * add split to dpo tests * fix rebase/merging error * handle edge case w logging * use accelerator for dpo datasets so it doesn't break the logger * missing args * validate checkpoint is an adapter for now * log warning when dataset strategy is not loadable --------- Co-authored-by: Agus <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR cleans up some hardcoding, improves the integration with trl's DPOTrainer and adds support for dpo prompt_strategies.